Search Results: "Christoph Berg"

17 May 2015

Lunar: Reproducible builds: week 3 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Tomasz Buchert submitted a patch to fix the currently overzealous package-contains-timestamped-gzip warning. Daniel Kahn Gillmor identified #588746 as a source of unreproducibility for packages using python-support. Packages fixed The following 57 packages became reproducible due to changes in their build dependencies: antlr-maven-plugin, aspectj-maven-plugin, build-helper-maven-plugin, clirr-maven-plugin, clojure-maven-plugin, cobertura-maven-plugin, coinor-ipopt, disruptor, doxia-maven-plugin, exec-maven-plugin, gcc-arm-none-eabi, greekocr4gamera, haskell-swish, jarjar-maven-plugin, javacc-maven-plugin, jetty8, latexml, libcgi-application-perl, libnet-ssleay-perl, libtest-yaml-valid-perl, libwiki-toolkit-perl, libwww-csrf-perl, mate-menu, maven-antrun-extended-plugin, maven-antrun-plugin, maven-archiver, maven-bundle-plugin, maven-clean-plugin, maven-compiler-plugin, maven-ear-plugin, maven-install-plugin, maven-invoker-plugin, maven-jar-plugin, maven-javadoc-plugin, maven-processor-plugin, maven-project-info-reports-plugin, maven-replacer-plugin, maven-resources-plugin, maven-shade-plugin, maven-site-plugin, maven-source-plugin, maven-stapler-plugin, modello-maven-plugin1.4, modello-maven-plugin, munge-maven-plugin, ocaml-bitstring, ocr4gamera, plexus-maven-plugin, properties-maven-plugin, ruby-magic, ruby-mocha, sisu-maven-plugin, syncache, vdk2, wvstreams, xml-maven-plugin, xmlbeans-maven-plugin. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Ben Hutchings also improved and merged several changes submitted by Lunar to linux. Currently untested because in contrib: reproducible.debian.net
Thanks to the reproducible-build team for running a buildd from hell. gregor herrmann
Mattia Rizzolo modified the script added last week to reschedule a package from Alioth, a reason can now be optionally specified. Holger Levsen splitted the package sets page so each set now has its own page. He also added new sets for Java packages, Haskell packages, Ruby packages, debian-installer packages, Go packages, and OCaml packages. Reiner Herrmann added locales-all to the set of packages installed in the build environment as its needed to properly identify variations due to the current locale. Holger Levsen improved the scheduling so new uploads get tested sooner. He also changed the .json output that is used by tracker.debian.org to lists FTBFS issues again but only for issues unrelated to the toolchain or our test setup. Amongst many other small fixes and additions, the graph colors should now be more friendly to red-colorblind people. The fix for pbuilder given in #677666 by Tim Landscheidt is now used. This fixed several FTBFS for OCaml packages. Work on rebuilding with different CPU has continued, a kvm-on-kvm build host has been set been set up for this purpose. debbindiff development Version 19 of debbindiff included a fix for a regression when handling info files. Version 20 fixes a bug when diffing files with many differences toward a last line with no newlines. It also now uses the proper encoding when writing the text output to a pipe, and detects info files better. Documentation update Thanks to Santiago Vila, the unneeded -depth option used with find when fixing mtimes has been removed from the examples. Package reviews 113 obsolete reviews have been removed this week while 77 has been added.

26 February 2015

Michael Banck: 26 Feb 2015

My recent Debian LTS activities

Over the past months, my employer credativ has sponsored some of my work time to keep PostgreSQL updated for squeeze-lts. Version 8.4 of PostgreSQL was declared end-of-life by the upstream PostgreSQL Global Development Group (PGDG) last summer, around the same time official squeeze support ended and squeeze-lts took over. Together with my colleagues Christoph Berg (who is on the PostgreSQL package maintainer team) and Bernd Helmle, we continued backpatching changes to 8.4. We tried our best to continue the PGDG backpatching policy and looked only at commits at the oldest still maintained branch, REL9_0_STABLE.

Our work is publicly available as a separate REL8_4_LTS branch on Github. The first release (called 8.4.22lts1) happened this month mostly coinciding with the official 9.0, 9.1, 9.2, 9.3 and 9.4 point releases. Christoph Berg has uploaded the postgresql-8.4 Debian package for squeeze-lts and release tarballs can be found on Github here (scroll down past the release notes for the tarballs).

We intend to keep the 8.4 branch updated on a best-effort community basis for the squeeze-lts lifetime. If you have not yet updated from 8.4 to a more recent version of PostgreSQL, you probably should. But if you are stuck on squeeze, you should use our LTS packages. If you have any questions or comments concerning PostgreSQL for squeeze-lts, contact me.

4 February 2015

Christoph Berg: apt.postgresql.org statistics

At this year's FOSDEM I gave a talk in the PostgreSQL devroom about Large Scale Quality Assurance in the PostgreSQL Ecosystem. The talk included a graph about the growth of the apt.postgresql.org repository that I want to share here as well: The yellow line at the very bottom is the number of different source package names, currently 71. From that, a somewhat larger number of actual source packages that include the "pgdgXX" version suffixes targeting the various distributions we have is built (blue). The number of different binary package names (green) is in about the same range. The dimension explosion then happens for the actual number of binary packages (black, almost 8000) targeting all distributions and architectures. The red line is the total size of the pool/ directory, currently a bit less than 6GB. (The graphs sometimes decrease when packages in the -testing distributions are promoted to the live distributions and the old live packages get removed.)

10 September 2014

Raphaël Hertzog: Freexian s first report about Debian Long Term Support

When we setup Freexian s offer to bring together funding from multiple companies in order to sponsor the work of multiple developers on Debian LTS, one of the rules that I imposed is that all paid contributors must provide a public monthly report of their paid work. While the LTS project officially started in June, the first month where contributors were actually paid has been July. Freexian sponsored Thorsten Alteholz and Holger Levsen for 10.5 hours each in July and for 16.5 hours each in August. Here are their reports: It s worth noting that Freexian sponsored Holger s work to fix the security tracker to support squeeze-lts. It s my belief that using the money of our sponsors to make it easier for everybody to contribute to Debian LTS is money well spent. As evidenced by the progress bar on Freexian s offer page, we have not yet reached our minimal goal of funding the equivalent of a half-time position. And it shows in the results, the dla-needed.txt still shows around 30 open issues. This is slightly better than the state two months ago but we can improve a lot on the average time to push out a security update To have an idea of the relative importance of the contributions of the paid developers, I counted the number of uploads made by Thorsten and Holger since July: of 40 updates, they took care of 19 of them, so about the half. I also looked at the other contributors: Rapha l Geissert stands out with 9 updates (I believe that he is contracted by lectricit de France for doing this) and most of the other contributors look like regular Debian maintainers taking care of their own packages (Paul Gevers with cacti, Christoph Berg with postgresql, Peter Palfrader with tor, Didier Raboud with cups, Kurt Roeckx with openssl, Balint Reczey with wireshark) except Matt Palmer and Luciano Bello who (likely) are benevolent members of the LTS team. There are multiple things to learn here:
  1. Paid contributors already handle almost 70% of the updates. Counting only on volunteers would not have worked.
  2. Quite a few companies that promised help (and got mentioned in the press release) have not delivered the promised help yet (neither through Freexian nor directly).
Last but not least, this project wouldn t exist without the support of multiple companies and organizations. Many thanks to them: Hopefully this list will expand over time! Any help to reach out to new companies and organizations is more than welcome.

One comment Liked this article? Click here. My blog is Flattr-enabled.

9 July 2014

Christoph Berg: New urxvt tab in current directory

Following Enrico's terminal-emulators comparison, I wanted to implement "start a new terminal tab in my current working directory" for rxvt-unicode aka urxvt. As Enrico notes, this functionality is something between "rather fragile" and non-existing, so I went to implement it myself. Martin Pohlack had the right hint, so here's the patch: 
--- /usr/lib/urxvt/perl/tabbed  2014-05-03 21:37:37.000000000 +0200
+++ ./tabbed    2014-07-09 18:50:26.000000000 +0200
@@ -97,6 +97,16 @@
       $term->resource (perl_ext_2 => $term->resource ("perl_ext_2") . ",-tabbed");
     ;
 
+   if (@  $self-> tabs   )  
+      # Get the working directory of the current tab and append a -cd to the command line
+      my $pid = $self-> cur pid ;
+      my $pwd = readlink "/proc/$pid/cwd";
+      #print "pid $pid pwd $pwd\n";
+      if ($pwd)  
+         push @argv, "-cd", $pwd;
+       
+    
+
    push @urxvt::TERM_EXT, urxvt::ext::tabbed::tab::;
 
    my $term = new urxvt::term
@@ -312,6 +322,12 @@
    1
  
 
+sub tab_child_start  
+   my ($self, $term, $pid) = @_;
+   $term-> pid  = $pid;
+   1;
+ 
+
 sub tab_start  
    my ($self, $tab) = @_;
 
@@ -402,7 +418,7 @@
 # simply proxies all interesting calls back to the tabbed class.
 
  
-   for my $hook (qw(start destroy key_press property_notify))  
+   for my $hook (qw(start destroy key_press property_notify child_start))  
       eval qq 
          sub on_$hook  
             my \$parent = \$_[0] term parent

25 June 2014

Christoph Berg: Comparing Version Numbers in Shell

On RedHat/CentOS/rpm systems, there's no dpkg --compare-versions available - sort -V can help to compare version numbers: 
version_lt ()  
    newest=$( ( echo "$1"; echo "$2" )   sort -V   tail -n1)
    [ "$1" != "$newest" ]
 
$ version_lt 1.5 1.1 && echo yes
$ version_lt 1.5 1.10 && echo yes
yes

16 May 2014

Christoph Berg: PostgreSQL 9.4 on Debian

Yesterday saw the first beta release of the new PostgreSQL version 9.4. Along with the sources, we uploaded binary packages to Debian experimental and apt.postgresql.org, so there's now packages ready to be tested on Debian wheezy, squeeze, testing/unstable, and Ubuntu trusty, saucy, precise, and lucid. If you are using one of the release distributions of Debian or Ubuntu, add this to your /etc/apt/sources.list.d/pgdg.list to have 9.4 available: deb http://apt.postgresql.org/pub/repos/apt/ codename-pgdg main 9.4 On Debian jessie and sid, install the packages from experimental. Happy testing!

25 February 2014

Christoph Berg: Trusty and Saucy on apt.postgresql.org

Over the past few weeks, new distributions have been added on apt.postgresql.org: Ubuntu 13.10 codenamed "saucy" and the upcoming Ubuntu LTS release 14.04 codenamed "trusty". Adding non-LTS releases for the benefit of developers using PostgreSQL on their notebooks and desktop machines has been a frequently requested item since we created the repository. I had some qualms about targeting a new Ubuntu release every 6 months, but with having automated more and more parts of the repository infrastructure, and the bootstrapping process now being painless, the distributions are now available for use. Technically, trusty started as empty, so it hasn't all packages yet, but of course all the PostgreSQL server packages are there, along with pgAdmin. Saucy started as a copy of precise (12.04) so it has all packages. Not all packages have been rebuilt for saucy, but the precise packages included (you can tell by the version number ending in .pgdg12.4+12 or .pgdg13.10+1) will work, unless apt complains about dependency problems. I have rebuilt the packages needing it I was aware about (most notably the postgresql-plperl packages) - if you spot problems, please let us know on the mailing list. Needless to say, last week's PostgreSQL server updates are already included in the repository.

12 February 2014

Christoph Berg: ci.debian.net on DDPO

More and more packages are getting autopkgtest aka DEP-8 testsuites these days. Thanks to Antonio Terceiro, there is ci.debian.net" running the tests. Last weekend, I've added a "CI" column on DDPO that shows the current test results for your packages. Enjoy, and add tests to your packages!

8 December 2013

Christoph Berg: TF101 flickering and a loose cable

My ASUS Transformer TF101 had suddenly started flickering in all sorts of funny colors some weeks ago. As tapping it gently on the table in the right angle made the problem go away temporarily, it was clear the problem was about a loose cable, or some other hardware connection issue. As I needed to go on a business trip the other day, I didn't look up the warranty expiration day until later that week. Then, Murphy struck: the tablet was now 2 years + 1 day old! Calling ASUS, some friendly guy there suggested I still tried to get ASUS to accept it for warranty, because the tablet had been with them last year for 5 days, so if they added that, it would still be within the warranty period. I filled out the RMA form, but one hour later the reply was they rejected it because it was out of warranty. Another guy on the phone then said they would probably only do the adding if it had been with them for maybe 10 days, or actually really 30 days, or whatever. Some googling suggested that the loose cable theory was indeed worth a try, so I took it apart. Thanks to a forum post I could then locate the display connector and fix it. Putting the case back together was actually harder than disassembling it because some plastic bits got stuck, but now everything is back to normal.

15 September 2013

Christoph Berg: Jessie, the HP 6715b, and Wifi

If you are upgrading your HP/Compaq 6715b to Debian Jessie, and suddenly Wifi stops working because the PCI device is gone, install the "rfkill" package: 
# lspci   tail -2
02:04.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 02)
10:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit Ethernet PCI Express (rev 02)
# rfkill list 1
1: hp-wifi: Wireless LAN
    Soft blocked: yes
    Hard blocked: no
# rfkill unblock wifi
# rfkill list 1
1: hp-wifi: Wireless LAN
    Soft blocked: no
    Hard blocked: no
# lspci   tail -2
10:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit Ethernet PCI Express (rev 02)
30:00.0 Network controller: Broadcom Corporation BCM4311 802.11a/b/g (rev 02)
Reports on the internet say that the same could be done by going into the BIOS and selecting "Reset to default" - this makes the Wifi LED active until about udev is started on the next boot. To be done: figure out how to automate this.

31 July 2013

Christoph Berg: Version Numbers

Following an idea by Ansgar Burchardt, I've done some digging on version numbers in Debian: Most common version numbers: 
projectb=> select version::text, count(*) from source group by 1 order by 2 desc;
  version     count 
------------+-------
 4:4.10.5-1     131
 1.0-1          120
 1.0.0-1         95
 1.1-1           95
 1.0.1-1         93
 1.2-1           88
 1.0-2           82
 0.2-1           80
 0.3-1           79
 0.5-1           77
 0.04-1          76
 1.1.1-1         76
 0.10-1          74
 1.4-1           72
 1.1-2           71
 0.1-1           70
 0.11-1          70
Version number with the most spellings: (considered equal by the dpkg definition, implemented in the "debversion" type) 
projectb=> select version::text, count(*) from source where version = '1.02-1' group by 1 order by 2 desc;
  version     count 
------------+-------
 1.2-1           88
 1.02-1          46
 1.002-1          4
 1.000002-1       1
 001.002-1        1
 1.00002-1        1
If we look at equivalent version numbers, the first table above looks entirely different: 
projectb=> select version, count(*) from source group by 1 order by 2 desc limit 30;
  version     count 
------------+-------
 0.3-1          162
 1.0-1          160
 0.05-1         156
 0.04-1         154
 0.02-1         151
 1.02-1         141
 0.006-1        133
 1.001-1        131
 4:4.10.5-1     131
 0.7-1          127
(I'm also participating in the "longest version number" contest, I've just uploaded bind9 version 1:9.8.4.dfsg.P1-6+nmu2+deb7u1~bpo60+1 to backports.)

25 March 2013

Christoph Berg: How not to monitor a boolean

We were lazy and wrote a simple PostgreSQL monitoring check in shell instead of using some proper language. The code looked about this: 
out=$(psql -tAc "SELECT some_stuff, t > now() - '1 day'::interval FROM some_table" some_db 2>&1)
case $out in
    *t) echo "OK: $out" ;;
    *) echo "NOT OK: $out" ;;
esac
If the string ends with 't', all is well, if it ends with 'f' or someting else, something is wrong. Unfortunately, this didn't go that well: OK: psql: FATAL: database "some_db" does not exist

7 February 2013

Christoph Berg: PostgreSQL minor releases

We've just put the new PostgreSQL minor releases live on apt.postgresql.org. Building 5 major versions for 10 distributions produces quite a lot of stuff: Compiling took a bit more than 10 hours on a 2-cpu VM. Of course that includes running regression tests and the postgresql-common testsuite. Note: This will be the last update published on pgapt.debian.net. Please update your sources.list entries to point to apt.postgresql.org!

7 December 2012

Christoph Berg: apt.postgresql.org

So we finally made it, and sent out an official announcement for apt.postgresql.org. This new repository hosts packages for all PostgreSQL server versions (at the moment 8.3, 8.4, 9.0, 9.1, 9.2) for several Debian/Ubuntu distributions (squeeze, wheezy, sid, precise) on two architectures (amd64, i386). Now add packages for extension modules on top of all these, and you get a really large amount of binaries from a small number of sources. Right now there's 1670 .deb files and 148 .dsc files, but the .dsc count includes variants that only differ in the version number per distribution (we attach .pgdg60+1 for squeeze packages, .pgdg70+1 for wheezy and so on), so the real number of different sources is rather something like 81, with 38 distinct source package names. Dimitri Fontaine, Magnus Hagander, and I have been working on this since I first presented the idea at PGconf.EU 2011 in Amsterdam. We now have a Jenkins server building all the packages, an archive server with the master repository, and a feed that syncs the repository to the postgresql.org FTP (well, mostly http) server. If you were previously using pgapt.debian.net, that's the same archive as on apt.postgresql.org (one rsync away). Please update your sources.list to point to apt.postgresql.org, I'll shut down the archive at that location at the end of January. Here's the Quickstart instructions from the Wiki page: Import the repository key from http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc: 
wget -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc   sudo apt-key add -
Edit /etc/apt/sources.list.d/pgdg.list. The distributions are called codename-pgdg. In the example, replace squeeze with the actual distribution you are using: 
deb http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg main
Configure apt's package pinning to prefer the PGDG packages over the Debian ones in /etc/apt/preferences.d/pgdg.pref: 
Package: *
Pin: release o=apt.postgresql.org
Pin-Priority: 500
Note: this will replace all your Debian/Ubuntu packages with available packages from the PGDG repository. If you do not want this, skip this step. Update the package lists, and install the pgdg-keyring package to automatically get repository key updates: 
apt-get update
apt-get install pgdg-keyring

30 November 2012

Christoph Berg: pgbouncer running on the same hardware

We have a PostgreSQL server with 16 cores that was apparently running well below its capacity: load something between 3.0 and 4.0, around 200 active database connections, almost all always being <IDLE>. However, when the tps count reached 7k transactions per second, things started to throttle, and pgbouncer (running on the database server) started listing up to half of the client connections to be in cl_waiting state. Load was still low, but application performance was bad. The culprit turned out to be the kernel scheduler, fairly distributing CPU time among all running processes. There's one single pgbouncer process, but hundreds of postgres processes. A simple renice of the pgbouncer process did the trick and gave us another extra 2k tps.

26 November 2012

Christoph Berg: Shared Memory and Swapping

We have this PostgreSQL server with plenty of RAM that is still using some of its swap over the day (up to 600MB). Then suddenly everything is swapped in again. It turned out the reason is there are two clusters running, and the second one isn't used as heavily as the first one. Disk I/O activity of the first cluster slowly evicts pages from the second shared buffers cache to swap, and then the daily pg_dump run reads them back every evening. I was not aware of an easy way to get numbers for "amount of SysV shared memory swapped to disk", but some googling led to shmctl(2): 
#define _GNU_SOURCE 1
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>
#include <unistd.h>
int main ()
 
        struct shm_info info;
        int max;
        long PAGE_SIZE = sysconf(_SC_PAGESIZE);
        max = shmctl(0, SHM_INFO, (struct shmid_ds *) &info);
        printf ("max: %d\nshm_tot: %ld\nshm_rss: %ld\nshm_swp: %ld\n",
                        max,
                        info.shm_tot * PAGE_SIZE,
                        info.shm_rss * PAGE_SIZE,
                        info.shm_swp * PAGE_SIZE);
        return 0;
The output looks like this: 
max: 13
shm_tot: 13232308224
shm_rss: 12626661376
shm_swp: 601616384
Update: Mark points out that ipcs -mu shows the same information. Thanks for the hint! 
# ipcs -mu
------ Shared Memory Status --------
segments allocated 2
pages allocated 3230544
pages resident  3177975
pages swapped   51585
Swap performance: 0 attempts     0 successes

22 November 2012

Christoph Berg: grep -r foobar

In Wheezy's grep version [1], you can omit the "." in 
$ grep -r foobar .
and just write 
$ grep -r foobar
[1] actually since 2.11

1 October 2012

Christoph Berg: mkfs.ext3 -b 1024

If you think you are smart and create an ext filesystem with 1024 bytes blocksize because there will be zillions of very small files, and then run into ENOSPC errors while there's both space and inodes left, you will probably see ext3_dx_add_entry: Directory index full! in the kernel log. Turns out that there's a limit of approximately 300,000 files per directory with 1k blocks, after which some hash tables are full. Recreate the filesystem with 2k blocks and the limit will be MUCH higher.

27 August 2012

Christoph Berg: PostgreSQL in Debian Hackathon

Almost a year has passed since my talk at pgconf.eu 2011 in Amsterdam on Connecting the Debian and PostgreSQL worlds, and unfortunately little has happened on that front, mostly due to my limited spare time between family and job. pgapt.debian.net is up and running, but got few updates and is lagging behind on PostgreSQL releases. Luckily, we got the project moving. Dimitri Fontaine and Magnus Hagander suggested to do a face-to-face meeting, so we got together at my house for two days last week and discussed ideas, repository layouts, build scripts, and whatnot to get all of us aligned for pushing the project ahead. My employer sponsored my time off work for that. We almost finished moving the repository to postgresql.org infrastructure, barring some questions of how to hook the repository into the existing mirror infrastructure; this should get resolved this week. The build server running Jenkins is still located on my laptop, but moving this to a proper host will also happen really soon now. We are using Mika Prokop's jenkins-debian-glue scripts for driving the package build from Jenkins. The big plus point about Jenkins is that it makes executing jobs on different distributions and architectures in parallel much easier than a bunch of homemade shell scripts could get us with reasonable effort. Here's a list of random points we discussed: We really aim at using unmodified packages from Debian as much as possible, and in fact this project doesn't mean to replace Debian's PostgreSQL packaging work, but to extend it beyond the number of server versions (and Debian and Ubuntu versions covered) supported. The people behind the Debian and Ubuntu packages, and this repository are mostly the same, so we will claim that "our" packages will be the same quality as the "original" ones. Big thanks go to Martin Pitt for maintaining the postgresql-common testsuite that really covers every aspect of running PostgreSQL servers on Debian/Ubuntu systems. Stay tuned for updates! :)

Next.

Previous.